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There is a rich and helpful body of academic literature related to writing and 
appraising qualitative research. This literature includes a discussion of what constitutes 
quality in qualitative research, as well as what contributes to a quality research report. In 
addition, there are several appraisal tools available to help those seeking to assess the 
quality of qualitative research reports. All of the Editors of The Qualitative Report think 
about how to evaluate qualitative research reports on a regular basis. We consider this 
issue both from a process perspective and a quality control perspective. In this paper, I 
describe how we as Editors of TQR came to the decision to develop our own appraisal 
tool for papers submitted to the journal and describe the tool that we developed—the 
TQR Rubric (Chenail, Cooper, Patron, & TQR Associates, 2011), as well as the 
developmental approach that guides our editorial processes and products. 

Quality in Qualitative Research 

Flick (2007) points out that the question of quality in qualitative research can be 
addressed on four different levels: the researcher interested in learning how well they 
have conducted their research, funding bodies seeking to detennine what studies should 
be funded or evaluating funded research, journal editors deciding which research reports 
to publish, and readers hoping to leam what research they can rely on in their own work. 
In this paper we address quality in qualitative research from the perspective of journal 
editors. As Flick notes, “Here, the quality issue is in some way doubled. Consideration 
of rigour and criteria in the research is seen as essential if the research is to be published. 
The research in its presentation has to be linked back to existing literature, for example— 
which is a criterion at the level of presentation” (p. 5). 

In the peer review process of reviewing manuscripts for journals, “a growing 
number of guidelines for assessing research papers (articles, proposals) are developed, 
used and published in different fields of application” (Flick, 2007, p. 22). For example, 
Kitto, Chesters, and Grbich (2008) describe how they assess the quality of submissions to 
the Medical Journal of Australia where their focus is on rigour of research and 
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transferability of findings. In addition to journals having their own preferences for 
manuscript quality, other groups have developed their own tools to appraise completed 
articles as part of systematic reviews of previously published work. The recent work by 
Hannes, Lockwood, and Pearson (2010) is one attempt to provide a comparison of three 
such appraisal instruments available online free of charge: the Joanna Briggs Institute 
(JBI) tool, the critical appraisal skills program (CASP) tool, and the evaluation tool for 
qualitative studies (ETQS). In addition to these qualitative research specific instruments, 
many journal editors use the Publication Manual of the American Psychological 
Association (2010) as a guide to the quality of a manuscript. However, as Polkinghome 
(2010) observes, the APA author guidelines were originally developed based on 
quantitative research reports, which can sometimes present a challenge to authors of 
qualitative research reports. 

In this environment of transparency regarding the articulation of what constitutes 
quality in qualitative research reporting, it is apparent that the context of the appraising 
body is an important factor in what the reviewer demarcates as quality and how these 
preferences are communicated to the general public as well as to potential and actual 
authors. In this spirit of localization and transparency, the Editors of The Qualitative 
Report embarked on a process of self-reflection of what constitutes quality in qualitative 
research writing from our local perspective and history. The goal of such an endeavor 
was first to make the TQR preferences overt to the journal’s internal community and 
second to produce a communication device though which these practices could be made 
more transparent. 


TQR Context 

Each academic journal has its own review process, editorial policies, and guiding 
philosophy. At The Qualitative Report, there are a few characteristics that we feel are 
central to our “brand”: a developmental approach, transparency, fostering a qualitative 
research community, and a focus on methodology. 

Developmental Approach 

All of the Editors of The Qualitative Report are committed to bringing a 
developmental approach to our work with both authors and editorial board members. 
Several years ago, this commitment led to the creation of the Manuscript Development 
Program (MDP), a review process that involves direction communication between 
authors and reviewers, and a team approach to supporting authors in developing their 
qualitative research reports until they are ready for publication. (For more infonnation on 
the MDP, please see our editorial statement at 
http ://www.nova, edu/ssss/QR/Editorial/editstm.html as well as A Guide for TQR Authors 
(Chenail, Wulff, & St. George, n.d.). In 2010, we introduced the Reviewer Development 
Program (RDP) to similarly support the professional development of our editorial board 
members. In this program, reviewers have the opportunity to advance within the editorial 
board based on quality and timeliness of reviews and become Assistant Editors and then 
Associate Editors. 
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In addition to the MDP and RDP, our developmental approach contributes to the 
training we provide our editorial board members. This includes providing new reviewers 
with appraisal tools and an annual in-person training session held at the annual TQR 
conference, conducting one-on-one training with new Assistant Editors regarding their 
role in our review process and how they can interface with the online manuscript tracking 
system we use, msTracker. We also send a quarterly editorial board newsletter to all 
members of the editorial board with tips for reviewing, policy updates, and other 
infonnation to help them in their editorial work. In addition, we solicit infonnation from 
our editorial board members regarding their accomplishments, publications, and 
promotions, and feature these achievements in the TQR News section of our website, as 
well as publishing them in our weekly newsletter, The Weekly Qualitative Report. 

Transparency 

Our commitment to transparency is evident in several key features and practices 
of The Qualitative Report. Perhaps most obvious is the fact that we are an open-access 
journal. Complete journal contents are available online free of charge. Besides our 
journal contents, our review process is a transparent review process, in which authors, 
reviewers, and editors communicate openly with one another, rather than employing a 
blind review process. This review process has been recognized by the Soros Foundation, 
which provided a grant to help us develop this open editorial process. In addition, all of 
our communications as an editorial board are available online for anyone interested in 
reading the editorial board quarterly newsletters and the minutes of the annual TQR 
editorial board meeting. We feel that authors and readers deserve to know the thinking 
guiding the editorial processes and decisions of the journal. 

Community-Building 

A third characteristic of the TQR “brand” is a focus on fostering a sense of 
community among the qualitative research community. The developmental approach and 
transparency described above contribute to building a sense of trust and openness among 
our community of authors, reviewers, editors, and readers. In the past few years, we have 
taken additional steps to develop the sense of a qualitative research community. In 2009, 
we started The Weekly Qualitative Report, a weekly newsletter available online and sent 
as an email to subscribers. The Weekly demonstrates that our commitment to qualitative 
researchers goes far beyond the bounds of The Qualitative Report. It includes listings of 
conferences and calls for papers associated with many other organizations. We feature 
current publications from many different journals that would report on qualitative studies. 
The Weekly also announces new issues of other qualitative research journals. We don’t 
see ourselves as in competition with these other journals, but as peers within a larger 
qualitative research community. The year after the launch of The Weekly, in January 
2010, we held the first TQR Conference, to provide a forum for qualitative research 
practitioners, teachers, and students to come together to share their discoveries and 
experiences related to practicing qualitative inquiry. The TQR 3 rd Annual Conference 
will take place in January 2012. All of these activities and products reflect our 
commitment to community-building. 
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Focus on Methodology 

A final feature of the TQR brand is a focus on methodology. Some academic 
journals that publish research reports prefer brief methods sections, as they wish to 
emphasize other aspects of the report, such as the theories informing the study and the 
study findings. At The Qualitative Report, we highlight the methodological details, 
asking authors to explain the reasoning behind design choices and to discuss the literature 
that guided those choices. We don’t publish papers that present theoretical arguments 
alone; articles published in TQR need to either be reporting on a particular qualitative 
research study or offering commentary and recommendations on a specific aspect of the 
practice of qualitative research. 

Theoretical Perspective 

As may be seen from the priorities and perspective shared above, we approach our 
work with The Qualitative Report from a philosophical and epistemological orientation 
rooted in the theory of social constructionism, focused on collaboration and a 
developmental emphasis. Perhaps it is no surprise that all of the TQR Editors are also 
Associates of the Taos Institute, “a community of scholars and practitioners concerned 
with the social processes essential for the construction of reason, knowledge, and human 
value” ( http://www.taosinstitute.net/) . 


TQR Rubric 


Development of the TQR Rubric 

As noted above, the Editors of The Qualitative Report have made it a priority to 
provide constructive, developmental support and mentoring to both authors and reviewers 
of qualitative research reports. There have been a variety of steps and tools that have 
been employed in this effort over the years. The Manuscript Development Program 
resulted in marginal comments providing targeted constructive feedback directly on the 
submitted manuscript. As the TQR Editors discovered that the same comments were 
proving to be helpful on paper after paper, the editors developed a document called TQR 
Ready Review Comments, so that editors could copy and paste (and modify as needed) 
these comments for authors, rather than composing them anew for each manuscript 
review. Initially this document was available only to the TQR Editors; however, in the 
spirit of both development and transparency, a revised and updated version of the TQR 
Ready Review Comments was made available to the entire editorial board in the Spring 
of 2011 (see Spring 2011 TQR Editorial Board Newsletter at 
http://www.nova.edu/ssss/QR/BoardNews/index.html . 

While the Ready Review comments have been very useful for the TQR editorial 
board, authors have not had access to these comments in advance of submission, unless 
they took it upon themselves to research the TQR editorial board newsletter archives, 
which is highly unlikely! Thus, authors have not been able to take these constructive 
suggestions into account as they develop their papers. We came to feel that it was more 
supportive of authors’ development to provide this type of information up front, so to 
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speak. In addition, the scale of submissions to the journal has been growing significantly 
as a result of the community-building initiatives such as the TQR Annual Conference and 
the development of The Weekly Qualitative Report. While we celebrate this increase in 
submissions, it has lead to our editorial board, especially our Editors, being rather 
overwhelmed by the quantity of manuscripts for which they are responsible. To aid 
authors and reduce the burden on reviewers and editors, we wanted to communicate as 
much helpful information to authors as possible prior to submission. In the past, we had 
made use of the CASP appraisal tool (see Chenail, 2011), but we had come to feel that it 
did not address all of the priorities encompassed by the TQR brand. All of these factors 
contributed to the decision to develop our own appraisal tool, designed to address the 
components of the TQR brand. 

To format our appraisal tool we selected a rubric which is “an instrument based 
on a set of criteria for evaluating student work” (Middle States Commission on Higher 
Education, 2007, p. 42). In our work as faculty members (e.g., Chenail, 2009), we use 
rubrics 


to make explicit, objective, and consistent the criteria for perfonnance that 
otherwise would be implicit, subjective, and inconsistent if a single letter 
grade were used as an indicator of perfonnance. Rubrics delineate what 
knowledge, content, skills, and behaviors are indicative of various levels 
of learning or mastery. Ideally, “grading” rubrics are shared with students 
before an exam, presentation, writing project, or other assessment activity. 
Conscious awareness of what he or she is expected to leam helps the 
student organize his or her work, encourages self-reflection about what is 
being learned and how it is being learned, and allows opportunities for 
self-assessment during the learning process. (Middle States Commission 
on Higher Education, 2007, p. 42) 

We have found the structure of the rubric to be one that allows us to present the key 
elements of an assignment in a simple and clear manner. The tool also helps us to 
communicate progress as students revise and resubmit their work. Lastly, rubrics, when 
shared with students at the beginning of a course, provide a self-assessment tool through 
which students can create, access, and revise their works before submitting them for 
formal review. For all of the reasons we decided to create our appraising tool in the form 
of a criterion-based rubric consisting of multiple sections enabling us to provide 
assessment and feedback in both quantitative and qualitative forms. 

Elements of the TQR Rubric 

The rubric that we have developed is divided into ten sections, which we refer to 
as “perfonnance areas” (please see Appendix to review entire TQR Rubric, Chenail, 
Cooper, Patron, & TQR Associates, 2011). Several of these performance areas are 
typical for checklists or appraisal tools for research reports, such as “Introductory 
Section,” “Literature Review,” “Methods Section,” and “Results Section,” while other 
perfonnance areas would appear to be unique to the TQR Rubric, such as a performance 
area for “Coherence” (see Appendix). Yet even in the “typical” performance areas, there 
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may be some atypical items, reflecting the unique priorities and brand of TQR. We 
highlight a few examples of this below and encourage you to refer to the complete TQR 
Rubric in the Appendix for more information. 

In keeping with our focus on methodology, we include criteria that both the title 
and key words for the paper should include a tenn indicating the qualitative research 
method used in the study. In addition, in the “Methods Section” performance area, we 
ask authors not only to provide a step-by-step description of the procedures used in the 
study, but also to include their rationale for each design choice, as well as an indication of 
the methodological literature they referred to in making those design choices. The 
“Methods Section” criteria also indicate that we would like authors to tell readers what 
constitutes data in their study, and to provide examples to illustrate each step of data 
analysis. 

In keeping with our emphasis on transparency, the TQR Rubric (Chenail, Cooper, 
Patron, & TQR Associates, 2011) includes an entire performance area on the “Role of 
Researcher.” The papers that we publish in the journal include an explicit description of 
the researcher’s context, interest in the topic under study, investment in the study, and 
intentions related to the study findings. Another way in which we encourage authors to 
be transparent is to include criteria such as “Makes clear who did what throughout study 
procedures” and “Active voice.” Anyone who has received a review in the Manuscript 
Development Program has likely received a request to shift from passive to active voice; 
we feel this is essential in being transparent about the role of the researcher(s). Another 
example of criteria to support transparency is a requirement that authors include within 
their paper a statement of the approval they received from an Institutional Review Board 
or other third party to assure that the safety and well-being of participants was addressed 
prior to the study being conducted. 


Discussion 

Multiple tools are available for authors and reviewers to assist in the evaluation of 
qualitative research reports. As Kitto, Chesters, & Grbich (2008) note, “In themselves, 
these criteria do not ensure rigour. However, they can strengthen rigour if they are used 
in concordance with a broader understanding of qualitative research design, data 
collection and analysis” (p. 243). The usefulness of any given appraisal tool is 
determined in large part by the objectives of the user. In our case, our objective for the 
TQR Rubric (Chenail, Cooper, Patron, & TQR Associates, 2011) is not just to assess the 
validity of a study’s findings, but to evaluate the quality of the report, where our criteria 
for quality reflect the TQR brand; and, to communicate our sense of rigor to authors and 
the larger qualitative research community. 

One of the priorities of TQR is transparency, and we recognize that we are not 
alone in valuing transparency. Flick (2007) observes that “transparency becomes 
relevant in several ways for enhancing the quality of qualitative research. Transparency 
means in general to make the research process, in its steps and in the decisions that 
influenced how data and results were produced, understandable to readers in the broadest 
sense” (p. 137). For us, transparency is not only important as a means of clear 
communication, and thus a measure of the trustworthiness of the findings, but 
transparency is also a reflection of the social constructionist perspective we bring to 
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reviewing qualitative research reports—a perspective which acknowledges that the 
results and the research report are constructions of the researcher. 

Regarding the use of checklists and rubrics, some people raise concerns about 
taking a one-size-lits-all approach to evaluating qualitative research (Barbour, 2001 cited 
in Flick, 2007). “The core of this problem is that the term ‘qualitative research’ is kind of 
an umbrella term. Under this umbrella, approaches assemble or are packed that have very 
different theoretical backgrounds, methodological principles, research issues and aims” 
(Flick, 2007, p. 6). Referring to the differences between grounded theory and 
conversation analysis, he then asks, “Is it possible then to evaluate these two examples 
with the same criteria once it comes to funding or publication, or do we need different 
criteria for each, taking the special features of both into account—without becoming 
completely relativistic in our judgments about good and bad research?” (Flick, 2007, p. 
7). We feel we address this concern in our emphasis on coherence; we are not asking that 
all research reports be written in the same manner, but that the results be reported in a 
way that aligns with the methodology indicated. Through our transparency and 
coherence in the TQR Rubric (Chenail, Cooper, Patron, & TQR Associates, 2011), we are 
asking authors do the same: Be transparent in communicating the choices you made to 
conceive, implement, and report qualitative research and show how these individual 
choices cohere with each other across the researching endeavor. 

Another concern raised regarding the use of appraisal instruments is that the 
instrument will end up detennining, and thus limiting, how research is reported (Barbour, 
2001, cited in Flick, 2007). As journal editors, we must confess we find this “limitation” 
less problematic. After all, we are indicating by means of the TQR Rubric (Chenail, 
Cooper, Patron, & TQR Associates, 2011) what we value and wish to publish in our 
journal; thus, we encourage authors to use the TQR Rubric as a developmental checklist 
to provide useful feedback throughout the writing process. While recognizing that this 
appraisal tool may reduce the creativity and range of reporting styles that end up being 
published in our journal, a focus on detailed explanations of methodological choices and 
rationales is part of the “TQR brand” and therefore, we see the rubric as a developmental 
tool to help authors be more transparent about their research rather than a limitation. 
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Appendix 

The Qualitative Report (TQR) Rubric 

Minimum 13 points required to enter TQR Manuscript Development Program (MDP) 


Total points out of 20: 


Author: 

| Title: | 

Performance Area and Criteria 

Non-Performance: 
includes none or 
minimal important 
elements of 
performance area 

(0 points) 

Partial: 

includes some but 
not all important 
elements of 
performance area 

(1 point) 

Complete: 
includes all (or 
almost all) important 
elements of 
performance area 
(2 points) 

Points: 

1. Opening Elements: 

A. Title is 12 words or less 

B. Title indicates most important elements of 
report, i.e., population, focus, methodology, 
and findings 

C. Abstract is 200 words or less 

D. Abstract reflects organizational structure of 
paper (i.e., presents problem/focus of study, 
research questions, participants, methodology, 
findings, key points from discussion of 
findings 

E. Paper includes Key Words 

F. Key Words include term for research method 

Elements needing 
attention: 

Elements needing 
attention: 

All important 
elements included. 

-or- 

Only the following 
element is missing: 


2. Introductory Section: 

A. Statement of research problem 

B. Statement of research objectives 

C. Indication of why local study has global 

importance 

D. Statement of rationale for study 

E. Naming of intended audience 

F. Indication of benefit of research (answers the 
“so what?” question) 

Elements needing 
attention: 

Elements needing 
attention: 

All important 
elements included. 

-or- 

Only the following 
element is missing: 


3. Literature Review: 

A. Offers synopsis of current literature on topic in 
terms of content and research processes used 

B. Demonstrates gap in literature re: content 
and/or research methods 

C. Explains how study will fill gap 

D. Provides reflections on literature vs. series of 
reports on sources 

E. Includes literature that helps define 
phenomenon shows what is known and not 
known about phenomenon 

F. Explains how literature led to research 
questions 

Elements needing 
attention: 

Elements needing 
attention: 

All important 
elements included. 

-or- 

Only the following 
element is missing: 


4. Role of Researcher: 

A. Describes researcher’s context, interest in topic 
and investment in study/intentions 

B. Makes clear who did what throughout study 
procedures 

C. Provides statement of IRB or other third-party 
approval secured to conduct study 

D. Describes how ethical issues were considered 
and addressed 

E. Describes how researcher bias was addressed 

F. Discusses steps taken to ensure rigor and 
trustworthiness of findings 

Elements needing 
attention: 

Elements needing 
attention: 

All important 
elements included. 

-or- 

Only the following 
element is missing: 


5. Methods Section: 

A. Explains how research design fits with research 
objectives 

B. Explains what type of qualitative inquiry was 
used 

C. Provides step by step description of 
procedures, with corresponding headings 

D. Describes sampling strategy and participant 
recruitment 

E. Explains steps of data generation, collection, 
and data analysis, as well as rationale for each 
design choice 

Elements needing 
attention: 

Elements needing 
attention: 

All important 
elements included. 

-or- 

Only the following 
element is missing: 
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F. Cites literature used to guide procedures 

G. Tells reader what constitutes data 

H. Provides examples to illustrate steps of data 
analysis 





6. Results Section: 

A. Tells reader how results will be organized 

B. Tells reader how results are derived from 
analysis 

C. Findings produced consistent with 
methodology indicated 

D. Presents exemplary evidence to support 
findings 

E. Explains how each excerpt supports 
assertions/findings 

F. Each excerpt illustrates unique qualitative 
distinction (rather than including multiple 
quotes to illustrate one finding) 

G. Presents demographic information of 
participants in composite fonn 

Elements needing 
attention: 

Elements needing 
attention: 

All important 
elements included. 

-or- 

Only the following 
element is missing: 


7. Discussion Section: 

A. Does not include discussion in results section 

B. Does not include findings in discussion section 

C. Does not repeat information already presented 
in paper 

D. Discusses how findings compare/contrast with 
what was known and/or not known in the 
literature 

E. Discusses limitations of study 

F. Discusses position on generalizability of 

results 

G. Discusses implications of findings 

H. Indicates area of future research 

I. Ends paper with discussion section 

Elements needing 
attention: 

Elements needing 
attention: 

All important 
elements included. 

-or- 

Only the following 
element is missing: 


8. References: 

A. Citations in text correspond to sources in 
reference list 

B. References are in APA style 

Elements needing 
attention: 

Elements needing 
attention: 

All important 
elements included. 

-or- 

Only the following 
element is missing: 


9. Writing: 

A. Effective use of headings 

B. Fluent English language 

C. Clear, precise writing 

D. Correct grammar and usage 

E. Avoids bias in language 

F. Strong mechanics of style 

G. Active voice 

H. Contextualized language reflects interpretive 
stance 

Elements needing 
attention: 

Elements needing 
attention: 

All important 
elements included. 

-or- 

Only the following 
element is missing: 


10. Coherence: 

A. Between title and abstract 

B. Between abstract and body of paper 

C. Between focus of study and literature reviewed 

D. Between research questions and methodology 

E. Between methodology presented and methods 
employed 

F. Between methodology and findings 

G. Between findings and research questions 

H. Between findings and stated implications 

Elements needing 
attention: 

Elements needing 
attention: 

All important 
elements included. 

-or- 

Only the following 
element is missing: 
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